Statistical Techniques for Text Classification Based on Word Recurrence Intervals

The decision as to whether two texts were written by the same author is usually a difficult one. Can an analysis of how the words in a text statistically cluster shed some light on authorship? In this paper we examine both English texts and the Greek source texts of the New Testament. The mathematical techniqes developed by Shannon [1,2] and Markov have been used for a number of years to analyse sequences of data, whether this be computer code, text, or DNA. These techniques and other probability-based techniques have enjoyed a large amount of usage in analysing DNA sequences [3] well as both written and spoken text [4,5]. Applications of linguistic methods to DNA sequence analysis have been explored by Dong and Searls [6] and others, and this is our motivation for exploring linguistic techniques for authorship (the corresponding problem in the field of DNA research is the phylogeny of organisms based on their DNA sequences). A seminal work in the area of authorship is Mosteller [7], a good overview of other work can be found in Oakes [8]. Durbin et al. [9] is a good reference of work done in analysing DNA sequences. Ortuño et al. [10] suggest using standard deviation of the ‘inter-word spacing’